Fall back to remote runtime on Spark Connect when the legacy namespace is unavailable#1464
Closed
sd-db wants to merge 2 commits into
Closed
Fall back to remote runtime on Spark Connect when the legacy namespace is unavailable#1464sd-db wants to merge 2 commits into
sd-db wants to merge 2 commits into
Conversation
…e is unavailable On a Databricks shared-access-mode (Spark Connect) cluster, importing databricks.sdk.runtime (which happens when WorkspaceClient.__init__ eagerly builds dbutils) materializes a legacy SparkContext via UserNamespaceInitializer.get_namespace_globals() and raises CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT (a PySparkRuntimeError, not ImportError). The surrounding 'except ImportError' does not catch it, so the error escapes the import and crashes WorkspaceClient construction. Treat a namespace-materialization failure the same as 'not in a classic runtime': log a warning and fall back to the existing OSS/remote implementation, which is Spark Connect-compatible (DatabricksSession + RemoteDbUtils). Fixes databricks#1463 Signed-off-by: Shubham Dhal <shubham.dhal@databricks.com>
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Contributor
|
Hi @sd-db — thanks for the detailed analysis and the fix. Carrying this forward upstream. Tests don't run on forked PRs, so the cleanest move was to land the same commit on an upstream branch. Opened #1469 with your commit preserved. Closing this in favor of #1469. Thanks again for the contribution! |
pull Bot
pushed a commit
to Future-Outlier/databricks-sdk-py
that referenced
this pull request
Jun 11, 2026
…e is unavailable (databricks#1469) ## Summary Importing `databricks.sdk.runtime` on a Spark Connect runtime (e.g. shared-access-mode clusters) no longer raises `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT` at import time. When the legacy user namespace cannot be materialized, the import now logs a warning and falls back to the existing Spark Connect-compatible remote implementation, so `WorkspaceClient()` construction succeeds on such clusters. Fixes databricks#1463. Carries forward @sd-db's work from databricks#1464 (closed because fork PRs in this repo cannot run tests). ## Why `WorkspaceClient.__init__` eagerly builds `dbutils` via `_make_dbutils`, which on a cluster does `from databricks.sdk.runtime import dbutils`. That import calls `UserNamespaceInitializer.getOrCreate().get_namespace_globals()`, materializing a legacy `SparkContext`. On a Spark Connect cluster this raises `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT` — a `pyspark.errors.PySparkRuntimeError`, not an `ImportError` — so the existing `except ImportError:` does not catch it and the error escapes the import, crashing `WorkspaceClient` construction before any API call. This is what databricks/dbt-databricks#1252 hits in Python models on shared clusters. The existing `except ImportError` branch is already the Spark Connect-compatible path (it builds `spark` via `DatabricksSession` and `dbutils` via `RemoteDbUtils`), so this PR routes the materialization failure there. A complementary follow-up — making `WorkspaceClient.dbutils` lazy via a `cached_property` so consumers that never touch it skip the build entirely — is noted in databricks#1463 as a separate discussion since it touches generated code. Related issue databricks#986 (off-cluster eager `RemoteDbUtils` auth failure) is the symmetric case and is intentionally not addressed here; the lazy-dbutils follow-up would unify both. ## What changed ### Behavioral changes On a Spark Connect runtime, importing `databricks.sdk.runtime` now logs a `WARNING` and uses the remote implementation instead of raising at import time. When `dbruntime` is absent (off-cluster) or the namespace materializes successfully (classic runtime), behavior is unchanged. ### Internal changes `databricks/sdk/runtime/__init__.py`: the runtime-namespace block is restructured into a single `try` with sibling `except ImportError` (existing — "not in a classic runtime") and `except Exception` (new — Spark Connect / `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT`, logged) clauses, plus an `if not _use_runtime_namespace:` guard over the existing — unchanged — OSS/remote block. The catch is intentionally broad rather than typed on `PySparkRuntimeError` to avoid pulling `pyspark` in at SDK import time just to narrow the exception type; the inline comment notes this. ## How is this tested? New `tests/test_runtime.py` simulates a Spark Connect runtime by injecting a fake `dbruntime` whose `get_namespace_globals()` raises `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT`, and asserts that: - reloading `databricks.sdk.runtime` survives the failure and falls back (`is_local_implementation is True`, `dbutils is not None`) - `WorkspaceClient(config=…)` constructs without raising — the direct reproduction of the reported failure Verified red→green locally. Full unit test suite (2098 tests) passes with no regressions. --------- Signed-off-by: Shubham Dhal <shubham.dhal@databricks.com> Signed-off-by: Divyansh Vijayvergia <171924202+Divyansh-db@users.noreply.github.com> Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Importing
databricks.sdk.runtimeon a Spark Connect runtime no longer raisesCONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT. When the legacy user namespace can't be materialized, the import now logs a warning and falls back to the existing Spark Connect-compatible remote implementation. This fixes the upstream crash behind dbt-databricks#1252.Fixes #1463
Why
WorkspaceClient.__init__eagerly builds dbutils via_make_dbutils, which on a cluster importsdatabricks.sdk.runtime. That import callsUserNamespaceInitializer.getOrCreate().get_namespace_globals(), materializing a legacySparkContext. On a shared-access-mode (Spark Connect) cluster this raisesCONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT— apyspark.errors.PySparkRuntimeError, not anImportError— so the existingexcept ImportError:doesn't catch it and the error escapes the import, crashingWorkspaceClientconstruction before any API call.Because the failure is in the constructor, consumers can't guard it with a
try/exceptaroundws.dbutils, and there's no config/env switch to skip the eager build — so downstream tools resort to monkey-patching the private_make_dbutils. Theexcept ImportErrorbranch is already the Spark Connect-compatible path (it buildssparkviaDatabricksSessionanddbutilsviaRemoteDbUtils), so this PR routes the materialization failure there.See #1463 for the full analysis. A complementary follow-up (making
WorkspaceClient.dbutilslazy) is noted there as a separate discussion since it touches generated code.What changed
Interface changes
None.
Behavioral changes
On a Spark Connect runtime, importing
databricks.sdk.runtimenow logs aWARNINGand uses the remote implementation instead of raising at import time. Whendbruntimeis absent, or when the namespace materializes successfully, behavior is unchanged.Internal changes
Restructured the runtime-namespace block in
databricks/sdk/runtime/__init__.pyinto a singletrywith siblingexcept ImportError(not a classic runtime) andexcept Exception(Spark Connect /CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT, logged) clauses, plus anif not _use_runtime_namespace:guard over the existing — unchanged — OSS/remote block.How is this tested?
New
tests/test_runtime.pysimulates a Spark Connect runtime by injecting a fakedbruntimewhoseget_namespace_globals()raisesCONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT, then asserts that (a) reloadingdatabricks.sdk.runtimesurvives and falls back (is_local_implementation is True,dbutils is not None) and (b)WorkspaceClient(config=…)constructs without raising — the direct reproduction of the reported failure. Verified red→green (both tests fail with the change reverted, pass with it). The full unit suite shows no regressions introduced by this change.